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1.  Introduction 


Classification  using  the  MetalMapper  advanced 
electromagnetic  sensor  was  demonstrated  at  the  former 
Pole  Mountain  Target  and  Maneuver  Area  (PMTMA),  WY 
in  2011.  This  report  summarizes  the  results  of  that 
demonstration.  The  document  Implementing  Classification  on 
Munitions  Response  Sites  (Ref.  1)  provides  practical 
information  for  deciding  whether  classification  is 
appropriate  to  a  particular  site  and  how  it  is  best 
implemented. 

Classification  is  motivated  by  the  need  to  perform 
munitions  response  more  cost-effectively  so  that  limited 
clean  up  dollars  can  be  used  to  reduce  real  risk  on 
munitions-contaminated  sites  sooner.  The  estimated 
liability  in  the  FY10  Defense  Environmental  Programs 
Report  to  Congress  for  Munitions  Response  is  $15.2B. 

(Ref.  2)  The  bulk  of  this  liability  is  $10. OB  for  the  1703 
sites  identified  in  the  Formerly  Used  Defense  Sites 
(FUDS)  program  and  $4.4B  for  the  2433  sites  identified 
on  Active  Installations.  The  remaining  $0.8B  is  in  Base 
Realignment  and  Closure  (BRAC).  The  estimated 
completion  dates  for  many  sites,  particularly  in  the  FUDS 
program,  are  decades  out  if  they  are  to  be  cleaned  up  at 
planned  funding  levels  using  current  practice. 

When  a  munitions  response  site  is  cleaned  up,  in  most 
cases,  it  is  mapped  with  a  geophysical  sensor  and  the 
locations  of  all  detectable  signals  are  excavated. 

Geophysical  sensors  detect  metal  and,  therefore,  many  of 
the  detections  do  not  correspond  to  munitions,  but  rather  to  harmless  metallic  objects.  Field 
experience  indicates  that  95-99%  or  more  of  objects  are  found  to  be  nonhazardous.  Current 
technology  does  not  provide  a  means  to  discriminate  between  munitions  and  other  items,  termed 
“clutter.”  As  a  result,  most  of  the  costs  to  remediate  a  munitions-contaminated  site  using  current 
methods  are  spent  on  excavating  targets  that  pose  no  threat. 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quality  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size  and  aspect  ratio.  The  values  of  these  parameters  may  then  be  used  to 
determine  whether  the  signal  arose  from  a  munition  or  harmless  clutter.  With  reliable  classification, 
only  the  munitions  need  to  be  removed  from  the  site. 


Pole  Mountain  Target  and  Maneuver 
Area ,  WY  —limited  variety  of 
munitions  used \  flat  and  open ,  minimal 
geologic  interference,  low  anomaly  density 

Munitions  —  37 -mm projectiles,  57 -mm 
projectile,  7 5 -mm projectiles,  3-in  stokes 
mortars 

Results  —  MetalMapper  was  used  to 
successfully  classify  all  of  the  targets  of 
interest  and  eliminate  up  to  90  %  of  the 
clutter.  A  production  contractor field 
crew  collected  high  quality  cued 
MetalMapper  data.  Both  production 
contractor  geophysicists  and  the 
developers  of  classification  methods  were 
successful  in  using  these  data  to  achieve 
substantial  classification.  Among  the 
production  geophysicists,  there  was  some 
variation  in  performance  but  even  the 
poorest  performer  was  able  to  correctly 
classify  2/ 3  of  the  clutter  while 
identifying  100  %  of  the  TOI. 
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The  Strategic  Environmental  Research  and  Development  Program  (SERDP)  and  the  Environmental 
Security  Technology  Certification  Program  (ESTCP)  have  supported  the  development  of  purpose- 
built  advanced  electromagnetic  sensors  and  associated  analysis  methods  for  classification.  Following 
the  successful  demonstration  of  classification  methods  in  controlled  test  environments,  ESTCP 
initiated  a  Classification  Pilot  Program  to  validate  the  application  in  real-world  conditions.  The  goal 
of  the  program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit 
approach,  based  on  principled  analysis  that  is  transparent  and  reproducible.  The  demonstrations  are 
planned  and  conducted  in  cooperation  with  regulators  and  program  managers  in  the  Services. 

The  physics  governing  the  electromagnetic  response  of  a  metal  object  is  well  understood  and 
predictable.  Data  collected  with  these  sensors  contain  the  same  information  content  on  any  site  and 
demonstrations  to  date  have  confirmed  that  classification  works  predictably.  Nevertheless, 
demonstrations  will  be  required  at  a  number  of  sites  to  represent  the  wide  variability  in  munitions 
types,  target  densities,  terrain,  vegetation,  geology,  land  use  history,  future  land  use,  and  other  site 
characteristics  that  will  affect  the  applicability  of  classification  and  to  establish  cost  effectiveness  and 
implementability.  The  demonstrations  also  present  an  opportunity  to  work  out  standard  operating 
procedures  and  establish  quality  control  (QC)  measures.  Prior  demonstrations  have  been  done  at 
the  former  Camp  Sibert,  AL,  the  former  Camp  San  Luis  Obispo,  CA,  the  former  Camp  Butner,  NC, 
and  the  former  Camp  Beale,  CA.  Details  about  past  and  ongoing  demonstrations  can  be  found  on 
the  SERDP-ESTCP  web  site  at  http:/ / serdp-estcp.org/Featured-Initiatives/Munitions-Response- 
Initiatives  /  Classification- Applied- to-Munitions-Response. 

The  demonstration  at  Pole  Mountain  continues  the  practice  of  production  geophysics  contractors 
collecting  and  analyzing  advanced  sensor  data  using  the  MetalMapper.  As  such,  one  purpose  of  the 
demonstration  was  to  train  production  contractors  using  the  MetalMapper  for  the  first  time.  This  is 
an  important  consideration  in  evaluating  and  applying  the  results.  We  discourage  potential 
customers  from  using  the  demonstration  results  to  rank  performers  and  make  contracting  selections; 
analysts  will  gain  experience  and  improve.  Data  were  also  analyzed  by  experienced  teams  from  the 
developers  of  the  classification  methods.  Table  1  shows  the  participants  and  their  roles  in  the 
MetalMapper  demonstration. 


Table  1.  Participants  in  the  MetalMapper  Demonstration  at  Pole  Mountain 


Task 

Performer(s) 

Task 

Performer(s) 

Site  Preparation 

URS 

MetalMapper  Data 
Analysis 

Parsons 

SAIC 

Sky  Research 

Shaw 

URS 

US  Army  Corps  of 
Engineers 

EM61-Mk2  Data 
Collection  and  Target 
Selection 

URS  (with  input  from 
ESTCP) 

MetalMapper  Data 
Collection 

Sky  Production 

Intrusive  Investigation 

URS 

Scoring 

Institute  for  Defense 
Analyses 
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2.  Pole  Mountain  Demonstration  Flow 

The  sequence  of  the  demonstration  is  outlined  in  the  flow  chart  in  Figure  2-1. 


Figure  2-1.  Flow  chart  outlining  steps  in  the  demonstration  at  Pole  Mountain.  Blue  boxes  are  tasks 
performed  by  ESTCP.  Others  are  tasks  performed  by  contractors. 

Prior  to  the  beginning  of  data  collection,  an  instrument  verification  strip  (IVS)  was  installed  and  the 
site  was  seeded  with  inert  munitions  and  small  industry 
standard  objects  (ISOs),  1-in  nominal  X  4-in  pipe  nipples. 

(Ref.  3)  Data  collectors  visited  the  IVS  twice  daily  to  verify 
equipment  function  at  the  start  and  end  of  each  day.  Since 
there  are  few  native  unexploded  ordnance  (UXO)  on  any 
munitions  response  site,  the  seeds  provided  sufficient 
targets  of  interest  (TOI)  to  allow  a  statistically  defensible 
determination  of  the  correct  classification  of  TOI. 

The  site  was  surveyed  with  an  EM61  to  provide  an  initial 
list  of  detected  anomalies.  The  MetalMapper  was  used  to 
collect  cued  data  over  each  anomaly.  All  detected  targets 
were  dug  up  to  provide  complete  ground  truth  for  the 


Targets  of  Interest  (TOI)  are  all  objects 
that  must  be  removed  from  the  site. 
Typically  the  TOI  will  include  all 
known  or  suspected  munitions  types,  any 
other  unexpected  munitions,  munitions 
parts  such  as  fu^es  that  present  an 
explosive  hazard,  and  all  seeded  items. 
When  classification  is  applied  to  a  site, 
the  local project  team  will  decide  what 
items  constitute  TOI. 


3 


purposes  of  determining  performance.  The  UXO  technicians  photographed  each  item  that  was  dug 
and  recorded  its  location,  depth,  and  description. 

The  geophysical  data  were  passed  to  the  data  analysis  teams.  A  complete  overview  of  the  analysis 
procedures  can  be  found  in  Ref  1.  Briefly,  the  analysts  used  methods  based  on  the  dipole  model  to 
estimate  target  parameters.  Analysts  were  offered  training  data  from  test  pit  measurements  and  the 
opportunity  to  request  additional  training  data  from  the  recovered  targets,  as  though  they  were 
doing  a  limited  number  of  sample  digs.  These  data  were  used  to  set  classifier  rules  —  the  decisions 
that  separate  the  anomalies  into  TOI  and  non-TOI.  The  classifiers  were  then  applied  to  all  of  the 
targets  that  remained  blind  for  each  demonstrator.  Since  training  data  was  by  request,  the  blind 
target  set  was  different  for  each  demonstration. 

The  product  required  from  each  analyst  was  a  ranked  anomaly  list  as  shown  in  Figure  2-2.  One  and 
only  one  judgment  was  required  for  each  entry  on  the  anomaly  list.  The  first  items  on  each  anomaly 
list  are  those  targets  for  which  reliable  parameters  cannot  be  extracted  and  therefore  must  be  dug. 
Next  are  those  items  which  the  analyst  is  the  most  confident  are  TOI.  The  items  are  ranked 
according  to  decreasing  likelihood  that  the  item  is  a  TOI.  Any  items  which  the  analyst  was  able  to 
analyze  but  was  not  able  to  make  a  classification  decision  on  at  this  time  were  placed  next  on  the 
anomaly  list.  Last  are  all  those  items  that  the  analyst  was  confident  are  not  TOI  ranked  by  their 
likelihood.  This  initial  list  is  shown  in  the  left  panel  of  Figure  2-2. 


Initial  Ranked  Anomaly  List 


Rank 

Anomaly 

ID 

Decision 

Statisitic 

Dig 

Comment 

1 

2498 

-9999 

Y 

Can’t  extract  reliable  parameters 

5 

811 

Y 

First 

313 

N 

Unable  to  classify 

Pass 

Threshold 

883 

N 

N 

N 

N 

High  likelihood  not  TOI 

N 

N 

N 

.02 

N 

.02 

N 

N 

.01 

N 

Final  Ranked  Anomaly  List 


Figure  2-2.  Initial  and  Final  Ranked  Anomaly  Lists.  A  detailed  description  is  in  the  text. 

The  seeds  were  divided  into  QC  seeds  and  blind  seeds.  When  analysts  submitted  their  initial 
prioritized  lists,  the  QC  seeds  were  used  to  provide  feedback  if  seed  targets  were  missed.  Analysts 
were  also  provided  with  the  ground  truth  information  on  all  anomalies  in  the  red  part  of  their  lists 
and  any  requested  anomalies  in  the  yellow  part.  This  is  signified  by  the  threshold  on  the  left  side  of 
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Figure  2-2.  Based  on  this  information,  the  analysts  were  then  allowed  to  revisit  their  rankings  and 
assignments  for  all  items  that  were  still  blind  until  they  were  satisfied  that  the  best  possible 
classification  had  been  achieved. 

In  the  final  list,  shown  in  the  right  panel  of  Figure  2-2,  the  analyst  was  required  to  provide  a 
threshold  that  corresponds  to  the  division  between  those  items  recommend  for  digging  and  those 
that  can  safely  remain  in  the  ground.  That  is,  the  list  is  all  red  and  green  with  a  threshold  separating 
the  two  categories.  The  final  prioritized  anomaly  lists  were  scored  against  the  emplaced  blind  seeds 
and  recovered  targets  by  IDA. 

3.  Site  Description  and  Preparation 

The  Pole  Mountain  Target  and  Maneuver  Area  is  a  62,448-acre  site  located  east  of  Laramie,  WY. 

The  PMTMA  was  established  in  1879  as  the  Fort  D.A.  Russell  Wood  and  Water  Reserve.  The  land 
status  alternated  between  national  forest  and  military  reservation  from  1897  to  1925.  The  Pole 
Mountain  area  has  also  been  known  as  the  Crow  Creek  Forest  Reserve,  Fort  D.A.  Russell  Target 
and  Maneuver  Range,  Fort  Francis  E.  Warren  Target  and  Maneuver  Range,  Pole  Mountain 
Reservation,  Pole  Mountain  Training  Annex,  and  Warren  Training  Annex.  It  was  extensively  used 
before  1959  as  a  target  and  maneuver  area  by  the  Army,  the  Reserve  Officers5  Training  Corps,  the 
Citizens’  Military  Training  Corps,  various  National  Guard  units,  and  the  Department  of  the  Air 
Force.  The  site  is  now  part  of  Medicine  Bow  National  Forest. 

There  are  several  Munitions  Response  Sites  at  the  Pole  Mountain  FUDS;  the  ESTCP  demonstration 
was  conducted  in  the  portion  of  the  site  referred  to  as  the  Bisbee  Hill  Maneuver  Area.  An  aerial 
photo  of  the  demonstration  area  is  shown  in  Figure  3-1  with  the  50-acre  demonstration  area  marked 
in  red. 

All  visible  metal  objects  were  removed  from  the  surface  at  the  site.  First  order  reference  points  were 
installed  by  a  registered  surveyor  for  geolocation  reference.  A  quiet  area  was  located  near  the 
demonstration  area  to  establish  an  instrument  verification  strip  (IVS)  used  for  daily  verification  of 
proper  sensor  operation  and  a  training  pit  to  collect  sensor  data  for  algorithm  training. 

A  variety  of  munitions  have  been  reported  as  used  at  PMTMA.  Physical  evidence  for  the  following 
items  was  discovered  during  the  recent  Remedial  Investigation  (Ref.  4): 

•  projectiles  containing  high  explosive  (HE)  filler  (37-mm  to  155-mm,  and  2.95-inch), 

•  shrapnel  projectiles  (75-mm  and  3-inch), 

•  3-inch  Stokes  mortars  (practice,  fuzed),  and 

•  60-mm  mortars  containing  HE  filler. 

The  objective  of  the  demonstration  was  to  detect  and  correctly  classify  all  TOI  on  the  site.  The 
analysts  were  provided  information  about  the  historical  use  and  known  munitions  types.  But,  the 
direction  specified  that,  in  addition  to  these  munitions,  any  unexpected  munitions  would  also  be 
considered  TOI. 
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UTM  Easting  (m) 


Figure  3-1.  Aerial  photo  of  part  of  the  Pole  Mountain  Target  and  Maneuver  Area  showing  the 

demonstration  site 


At  a  live  site,  the  number  of  targets  of  interest  is  small,  far  from  enough  to  determine  any 
demonstrator’s  classification  performance  with  acceptable  statistical  confidence  bounds.  In  fact,  on 
the  portion  of  the  Pole  Mountain  site  used  for  this  demonstration,  no  UXO  were  recovered  in  the 
intrusive  investigation.  Therefore,  the  site  was  seeded  with  enough  TOI  to  ensure  statistical  validity 
on  measures  of  classification  of  targets  of  interest.  The  TOI  seeds  are  listed  in  Table  3-1.  The 
seeds  included  not  only  inert  munitions,  but  also  industry  standard  objects.  (Ref.  3)  The  ISOs  are 
also  considered  TOI  and  expected  to  be  both  detected  and  correctly  classified. 


Table  3-1.  Seeds  Emplaced  for  the  Pole  Mountain  demonstration 


Item 

Number 

Depth  Range  (cm)* 

Industry  Standard  Object  -  Small 

40 

15-25 

37-mm  projectile 

43 

15-30 

57-mm  projectile 

10 

20-35 

60-mm  mortar 

41 

30 

75-mm  projectile 

25 

20-40 

3-in  stokes  mortar 

1 

30 

*Depths  are  to  the  center  of  the  object. 
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No  attempt  was  made  to  separate  the  seeds  from  the  surrounding  clutter.  For  safety,  seeds  were 
emplaced  using  standard  anomaly  avoidance  procedures.  For  realism,  the  emplacement  teams  were 
instructed  to  replace  any  metal  dug  up  during  emplacement  back  in  the  hole  with  the  seeded  object. 

4.  EM61  Detection  Survey 

An  initial  survey  was  performed  with  an  EM61-MK2  in  its  standard  cart  configuration  with  cm-level 
global-positioning-system  (GPS)  navigation.  These  data  were  used  both  to  provide  a  common 
anomaly  list  for  the  MetalMapper  data  collection  that  followed  and  to  attempt  classification  using 
only  the  EM61  data  as  a  point  of  comparison. 

The  data  quality  objectives  for  the  detection  survey  were  based  on  the  37-mm  projectile,  which  was 
expected  to  be  the  most  difficult  to  detect  TOI  at  the  site.  The  EM61  survey  was  performed  on 
half-meter  line  spacing.  The  anomaly  selection  criteria  were  set  to  detect  a  37-mm  projectile  at  a 
depth  of  one  foot  (30  cm).  This  depth  was  chosen  as  the  deepest  depth  to  which  a  37-mm  could  be 
reliably  detected. 

The  EM61  signal  strength  in  channel  2  versus  depth  for  a  37-mm  projectile  is  shown  in  Figure  4-1. 
(Ref.  5)  The  signal  at  30-cm  depth  for  the  least  favorable  orientation  is  5.2  mV,  which  was  used  as 
the  amplitude  threshold  for  identifying  anomalies.  This  threshold  corresponds  to  detection  of  all 
60-mm  mortars  to  60  cm,  all  75-mm  projectiles  to  80  cm,  and  all  3-in  stokes  mortars  to  85  cm. 

Depth  (cm  bgs) 

0  20  40  60  80 


Figure  4-1.  EM61  response  in  Gate  2  versus  depth  for  a  37-mm  projectile 

The  EM61  survey  resulted  in  a  total  of  2370  anomalies  in  the  50-acre  demonstration  area,  including 
the  seeds.  This  translates  to  approximately  50  anomalies  per  acre.  Little  geologic  response  was  seen 
in  the  EM61  data  and  in  the  subsequently  collected  MetalMapper  data.  Details  can  be  found  in  the 
vendor’s  reports.  (Ref.  6,  7)  All  seeds  were  detected  in  the  EM61  survey  using  the  threshold 
described  above. 
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5.  MetalMapper  Data  Collection 

The  MetalMapper  developed  by  Geometries  is  designed  to  be  a  stand-alone  survey  and  cued 
detection  system.  The  system,  shown  in  Figure  5-1,  is  composed  of  three  orthogonal  1-m  x  1-m 
transmitters  for  target  illumination  and  7  three-axis  receivers  for  recording  the  response.  Its 
sampling  is  electronically  programmable  and  therefore  flexible.  It  measured  the  decay  curve  up  to 
8  ms  after  the  transmitters  were  turned  off.  It  was  deployed  in  a  sled  configuration  mounted  to  an 
all-terrain  vehicle.  Centimeter-level  GPS  is  used  for  navigation  and  geolocation  and  an  inertial 
measurement  unit  (IMU)  is  used  to  measure  platform  orientation.  In  cued  mode,  MetalMapper  is 
positioned  over  each  anomaly  on  its  target  list  and  collects  the  full  suite  of  data  while  stationary.  The 
digital  data  set  produced  by  MetalMapper  is  fully  described  in  Ref.  8. 


Figure  5-1.  Schematic  and  photo  of  the  MetalMapper  as  used  at  Pole  Mountain 

In  this  demonstration  MetalMapper  was  used  only  in  cued  mode.  Because  of  site  limitations,  only 
one  data  collection  was  possible;  the  production  arm  of  Sky  Research  collected  MetalMapper  data  at 
Pole  Mountain.  Details  on  the  data  collection  and  QC  procedures  for  this  team  can  be  found  in 
their  report.  (Ref.  7)  The  most  common  QC  failure  was  that  the  MetalMapper  was  positioned  too 
far  from  the  anomaly  to  obtain  reliable  parameter  estimates.  If  the  separation  between  the  center  of 
the  MetalMapper  and  the  anomaly  location  was  more  than  40  cm,  the  anomaly  was  revisited  the  next 
day  and  additional  data  collected  within  the  40-cm  specification. 

Excluding  the  first  and  last  day’s  field  work,  this  data  collection  team  averaged  215  cued  anomalies 
per  day  (32  per  hour).  They  required  125  QC  recollects,  corresponding  to  a  little  over  5%  of  the 
anomalies  measured. 

6.  Analysis  of  MetalMapper  Data 

The  MetalMapper  data  were  analyzed  by  multiple  analysts,  including  both  the  developers  of  the 
analysis  methods  and  production  geophysics  vendors.  Figure  6-1  shows  an  overview  of  the  results 
achieved  by  all  analysts  working  with  the  MetalMapper  data.  The  panel  on  the  left  shows  the 
percent  of  TOI  correctly  classified  versus  the  number  of  clutter  at  each  analyst’s  operating 
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Figure  6-1.  Results  of  all  analysts  in  the  Pole  Mountain  demonstration.  The  left  panel  shows  the 
percent  TOI  correctly  identified  versus  number  of  clutter  at  the  analysts  chosen  threshold.  The  right 
panel  shows  the  number  of  clutter  that  would  have  to  be  dug  to  classify  100%  of  the  TOI,  regardless 

of  where  the  analyst  put  the  threshold. 

threshold.  The  inset  expands  the  upper  left  of  the  graph  for  clarity.  Desired  performance  is  to 
correctly  classify  100%  of  the  TOI  and  eliminate  all  of  the  clutter.  The  panel  on  the  right  shows  the 
number  of  clutter  that  the  analyst  needed  to  dig  to  get  to  100%  correct  classification  of  the  TOI, 
regardless  of  where  the  analyst  put  the  threshold.  This  can  be  thought  of  as  the  best  the  analyst 
could  have  done  by  putting  the  threshold  in  exactly  the  optimum  place  where  the  last  TOI  is  found 
in  the  ranked  list.  The  symbols  above  the  bars  correspond  to  the  symbols  in  the  left  panel. 

At  their  specified  thresholds,  most  analysts  achieved  very  good  results,  correctly  classifying  100%  of 
the  TOI  and  eliminating  more  than  80%  of  the  clutter.  This  includes  both  the  classification 
algorithm  developers  and  the  production  contractors  and  is  quite  remarkable,  especially  considering 
that  this  was  the  first  attempt  at  analyzing  MetalMapper  data  by  some  of  these  analysts 

The  range  in  performance  is  shown  in  the  right  panel.  The  best  performer  could  have  eliminated  all 
but  50  of  the  clutter  (a  97%  reduction)  with  100%  correct  classification  of  TOI,  where  as  the 
poorest  performer  at  best  could  have  eliminated  about  only  66%  of  the  clutter. 


6.1  Analysis  by  Production  Geophysicists 

The  focus  in  this  demonstration  was  on  the  performance  of  production  geophysics  teams  so  we 
present  representative  examples  of  the  results  from  these  analysts.  A  complete  compendium  of  all 
results  can  be  found  in  the  report  by  IDA.  (Ref.  9) 

Geophysicists  from  the  production  arm  of  Sky  Research  analyzed  the  MetalMapper  data  collected  by 
their  field  colleagues  using  methods  developed  by  the  research  geophysicists  at  Sky.  (Ref.  7)  The 
results  of  this  analysis  are  shown  in  Figure  6-2.  The  colors  on  the  plot  correspond  to  the  red  and 
green  colors  in  the  final  ranked  anomaly  list  as  shown  in  Figure  2-2.  The  red  are  the  items  the 
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Figure  6-2.  Results  of  the  Sky  production  team  analysis  of  their  MetalMapper  data 


analyst  classified  as  “high  likelihood  TOI”  and  the  green  are  those  the  analyst  called  “high  likelihood 
not  TOI.”  No  anomalies  were  classified  as  “unable  to  classify.”  The  graph  plots  the  percent  of  the 
targets  of  interest  correctly  classified  on  the  vertical  axis  and  the  number  of  clutter  items  on  the 
horizontal  axis.  The  offset  from  zero  in  the  starting  point  reflects  any  training  data  that  the  analyst 
requested.  Any  anomalies  classified  as  “can’t  extract  reliable  parameters”  would  be  represented  by 
an  initial  black  line;  no  anomalies  were  in  this  category  for  this  analysis.  The  blue  dot  represents  the 
threshold  selected  by  the  analyst  and  the  orange  dot  shows  the  point  on  the  ranked  anomaly  list 
where  100%  of  the  target  of  interest  are  captured.  Ideally,  a  classifier  would  correctly  identify  all 
targets  of  interest  in  the  red  with  zero  clutter  and  all  of  the  clutter  would  be  in  the  green.  In  this 
case,  the  red  part  of  the  curve  would  go  straight  up  to  100%  and  the  green  part  of  the  curve  would 
run  straight  across  the  top  axis.  Success  in  these  demonstrations  was  defined  by  eliminating  the 
maximum  amount  of  clutter  while  correctly  identifying  all  of  the  TOI. 

In  this  demonstration,  there  were  2208  total  clutter  items  as  determined  from  the  ground  truth. 

This  analyst  was  able  to  correctly  identify  almost  2000  of  these  items  at  their  threshold,  for  a 
possible  savings  of  more  than  90%  of  the  digs 

Several  groups  analyzed  the  Pole  Mountain  MetalMapper  data  using  the  UX- Analyze  module  of 
Oasis  montaj.  Figure  6-3  shows  the  results  of  the  Parsons  analysis.  (Ref  10)  This  analyst  labeled  a 
significant  number  of  the  anomalies  as  “can’t  extract  reliable  parameters”  as  shown  by  the  black 
points  in  Figure  6-3.  Only  one  of  these  anomalies  corresponded  to  a  target  of  interest.  Of  the 
anomalies  for  which  parameter  extraction  was  successful,  these  results  show  the  right  general  trend, 
in  that  the  red  part  of  the  curve  rises  steeply  initially  and  more  than  75%  of  the  clutter  anomalies  are 
correctly  classified  as  not  TOI. 
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Figure  6-3.  Results  of  the  Parsons  analysis  of  the  Pole  Mountain  MetalMapper  data 


A  US  Army  Corps  of  Engineers  analyst  also  used  UX- Analyze  and  achieved  the  results  shown  in 
Figure  6-4.  (Ref.  11)  This  analysis  is  very  efficient  at  the  beginning  of  the  ranked  anomaly  list;  the 
ROC  curve  is  almost  vertical.  Identifying  the  last  25  TOI  required  a  significant  number  of  clutter 
digs.  Although  the  last  two  TOI  identified  were  a  37-mm  projectile  and  an  ISO,  the  25  more 
difficult  TOI  comprised  all  possible  types.  This  analyst  was  able  to  correctly  classify  more  than  80% 
of  the  clutter  as  not  TOI. 
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Figure  6-4.  Results  of  a  USACE  geophysicist’s  analysis  of  the  Pole  Mountain  MetalMapper  data 


A  hybrid  approach  was  used  by  analysts  from  URS.  (Ref.  12)  These  geophysicists  used  UX- Analyze 
to  estimate  target  parameters  from  the  MetalMapper  data  then  used  an  Artificial  Neural  Net  to  order 
the  anomaly  list.  The  ROC  curve  that  results  from  this  analysis  is  shown  in  Figure  6-5.  After 
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requesting  46  training  digs  and  declaring  51  anomalies  as  “can’t  extract  reliable  parameters,”  this 
analyst  was  able  to  correctly  classify  almost  90%  of  the  clutter  as  non-hazardous. 
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Figure  6-5.  Results  of  the  URS  hybrid  analysis  of  the  Pole  Mountain  MetalMapper  data 


Finally,  geophysicists  from  Shaw  worked  with  algorithm  developers  from  Sky  Research  on  an 
analysis  of  the  MetalMapper  data.  (Ref.  13)  The  ROC  curve  that  results  from  this  analysis  is  shown 
in  Figure  6-6.  These  analysts  only  required  96  clutter  digs  to  identify  100%  of  the  targets  of  interest, 
meaning  they  were  able  to  correctly  classify  more  than  95%  of  the  clutter  as  nonhazardous. 
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Figure  6-6.  Results  of  the  analysis  by  Shaw  geophysicists  working  with  one  of  the  algorithm 

developers 
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7.  Cost  Comparison 

The  demonstration  took  place  on  a  small  part  of  the  Pole  Mountain  site  and  incurred  costs  for  many 
items  specific  to  a  demonstration  that  would  not  be  needed  in  an  application  of  classification  to  a 
real  site.  Nevertheless,  we  can  extract  meaningful  projected  performance  for  the  technology  and 
apply  reasonable  industry  unit  costs  for  various  elements  to  arrive  at  a  total  cost  comparison  for 
clearing  an  example  500-acre  site  with  and  without  the  use  of  classification. 

We  made  the  following  assumptions: 

•  The  example  takes  place  in  an  area  with  similar  munitions  types  and  the  same  density  of 
anomalies  as  seen  in  the  demonstration.  Excluding  the  seeds,  there  were  2208  anomalies  in 
the  50-acre  demonstration  area.  All  of  these  were  clutter.  Extrapolating,  we  would  expect 
about  22,100  anomalies  in  a  similar  500-acre  area. 

•  Although  no  TOI  were  found  in  the  small  demonstration  area  at  Pole  Mountain,  we  assume 
a  small  number,  50,  will  be  found  in  a  larger  production  operation,  and  22,050  clutter  items. 

•  The  baseline  is  an  EM61  survey  with  0.5-m  line  spacing.  This  would  be  used  to  select 
anomalies  for  digging  without  classification  and  the  same  anomalies  would  be  interrogated 
with  MetalMapper  and  classified. 

•  The  site  is  seeded  at  a  rate  so  on  average  one  seed  will  be  encountered  each  day  of 
MetalMapper  data  collection.  With  an  estimate  of  22,100  total  anomalies  and  a  production 
rate  of  200  anomalies  per  day,  we  seed  a  conservative  125  inert  items.  These  QC  seeds 
would  be  used  whether  classification  was  used  on  the  site  or  not. 

•  The  classification  performance  is  as  achieved  by  the  Sky  production  team  or  URS,  as  shown 
in  Figures  6-2  and  6-5,  with  ~90%  of  the  clutter  correctly  identified  and  remaining  undug. 

•  The  unit  costs  are  as  shown  in  Table  8-1. 


Table  8-1.  Unit  cost  assumptions 


Item 

Units 

Cost 

Digs 

Per  hole 

$125 

MetalMapper  Classification 

Per  anomaly 

$25  and  $35 

EM61  Survey  Data  Collection  and  Analysis 

500  acres 

$381,000 

Seed  Emplacement 

125  seeds 

$22,650 

With  these  assumptions  the  costs  were  calculated  using  the  elements  shown  in  Table  8-2.  If 
classification  can  be  done  for  $35  per  anomaly  including  both  data  collection  and  analysis,  which  is 
consistent  with  the  projections  of  the  production  companies  based  on  this  demonstration,  a  54% 
overall  savings  is  possible.  If  this  cost  can  be  lowered  to  $25  to  classify  each  anomaly,  the  potential 
project  savings  increases  to  61%. 
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Table  8-2.  Cost  Comparison  for  500  acres  of  comparable  Pole  Mountain  site 


Item 

j  No  Classification  | 

[  Classification 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

($35  per  MM) 

Cost/$ 

($25  per  MM) 

Seeds 

125  items 

22,650 

125  items 

22,650 

22,650 

EM61  Survey 

500  acres 

381,000 

500  acres 

381,000 

381,000 

MetalMapper 

Classification 

n/a 

0 

22,225 

anomalies 

777,875 

555,625 

Seeds  Dug 

125 

15,625 

125 

15,625 

15,625 

Native  UXO  Dug 

50 

6,250 

50 

6,250 

6,250 

Clutter  Dug 

22,050 

2,756,250 

2205 

275,625 

275,625 

TOTAL 

3,181,775 

1,479,025 

1,256,775 

Percent  Savings 

54% 

61% 

8.  Conclusions 

Classification  was  used  on  the  Pole  Mountain  Target  and  Maneuver  Area  to  successfully  identify  all 
of  the  TOI  and  eliminate  about  90%  of  the  clutter.  A  production  contractor  field  crew  from  Sky 
collected  high  quality  cued  MetalMapper  data.  Both  production  contractor  geophysicists  and  the 
developers  of  classification  methods  were  successful  in  using  these  data  to  achieve  substantial 
classification.  Among  the  production  geophysicists,  there  was  some  variation  in  performance. 
Additional  training  is  needed  so  that  all  analysts  are  able  to  identify  problems  that  can  be 
encountered  in  the  multiple  analysis  steps  required  in  classification. 

The  Pole  Mountain  site  is  not  very  challenging  for  classification  using  an  advanced  EMI  sensor.. 
Even  though  the  targets  of  interest  included  small  37-mm  projectiles  as  well  as  seeded  small  ISO’s, 
the  site  geologic  background  was  very  low  resulting  in  high  signal-to-noise  data  in  most  cases.  The 
anomaly  density  was  low,  at  about  50  per  acre,  and  did  not  present  any  particular  difficulty. 


9.  Acronyms 


BRAC 

ESTCP 

FUDS 

GPS 

IMU 

ISO 

IVS 

QC 

RMS 

SERDP 

TOI 

UXO 


Base  Realignment  and  Closure 

Environmental  Security  Technology  Certification  Program 

Formerly  Used  Defense  Site 

Global  Positioning  System 

Inertial  Measurement  Unit 

Industry  Standard  Object 

Instrument  Verification  Strip 

Quality  Control 

Root  Mean  Square 

Strategic  Environmental  Research  and  Development  Program 
Target  of  Interest 
Unexploded  Ordnance 
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